Introduction to Sequence Modeling: Data, Order, and Temporal Dynamics

In the realm of Artificial Intelligence, sequence modeling shifts the focus from static snapshots to temporal streams. Standard machine learning tasks often assume that data points are Independent and Identically Distributed (IID), meaning the order of samples does not influence the outcome.

Sequence modeling explicitly rejects this, focusing on three core pillars:

Violation of Permutation Invariance: In tabular data, the column order is arbitrary. In sequences, order is the primary feature. Swapping "The cat ate the rat" to "The rat ate the cat" fundamentally changes the semantic ground truth despite identical tokens.
Autoregressive Properties: We assume that an observation at time $t$ is mathematically conditioned on its history ($t-1, t-2, \dots, 1$). This necessitates transition probabilities to capture how information evolves.
Variable-Length Mapping: Unlike fixed 28x28 pixel grids, sequences like sentences or seismic waves are elastic. Models must process inputs of length $N$ and produce outputs of length $M$ using consistent parameters.

QUESTION 1

Which of the following best describes why a standard Feed-Forward Neural Network (MLP) struggles with sequence modeling?

It cannot handle backpropagation.

It assumes data points are Independent and Identically Distributed (IID) and requires fixed-length inputs.

It is incapable of using activation functions like ReLU.

It is too computationally expensive for text data.